Mladenov Marijana‘ UDK: 159.922:004.8(4-672EU) 


http://orcid.org/0000-0002-45 74-5159 Original scientific paper 
DOI: 10.5937/ptp2300032M 
Received: 24.01.2023. 
Approved on: 28.02.2023. 
Pages: 32—43 


HUMAN VS. ARTIFICIAL INTELLIGENCE 
— EU’S LEGAL RESPONSE 


ABSTRACT: Artificial intelligence (AI) has the capacity to improve 
not only the individual quality of life, but also economic and social 
welfare. Although the AI systems have many advantages, they also pose 
significant risks, creating a wide range of moral and legal dilemmas. The 
European Union has been creating a legal framework for developing, 
trading, and using AI-driven products, services, and systems to reduce the 
risks connected with the AI systems and to prevent any possible harm they 
may cause. The main focus of this paper refers to the analysis of the Proposal 
for the Artificial Intelligence Act submitted by the European Commission 
in April 2021. The goal of the article is to move toward a possible resolution 
to the dilemma of whether the AIA proposal is appropriate for the AI era 
by addressing the scope of its application, the prohibited AI practices, 
rules on high-risk AI systems, specific transparency obligations, as well as 
certain regulatory gaps. The article should be viewed as an initial analysis 
of the AIA proposal in order to provide a useful framework for the future 
discussion. 


Keywords: artificial intelligence, the Europan Union, regulatory 
framework, the Proposal for the Artificial Intelligence Act. 


“ LLD, Associate professor, Faculty of Law for Commerce and Judiciary in Novi Sad, The University 
of Business Academy in Novi Sad, Republic of Serbia, e-mail: alavuk@pravni-fakultet.info 
(OBO 2023 by the authors. This article is an open access article distributed under the terms and 
conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/ 
licenses/by/4.0/). 


32 


HUMAN VS. ARTIFICIAL INTELLIGENCE — EU’S LEGAL RESPONSE 


1. Introduction 


Artificial intelligence (AI) has transformed many industries in recent 
years and still attracts global headlines (Perucica& Andjelkovic, 2021. p.348). 
AI has the capacity to improve not only the individual quality of life but also 
economic and social welfare (Kolarevi¢, 2022, p.111). However, while AI 
systems have many advantages, they also pose significant risks, creating a 
wide range of moral and legal dilemmas (Bjelajac& Filipovski, 2021. p.11). 

The European Union has been creating a legal framework for developing, 
trading, and using Al-driven products, services, and systems to reduce the 
risks connected with AI systems and to prevent any possible harm they may 
cause. The European Parliament passed a “Resolution on Civil Law Rules on 
Robotics” on February 16, 2017, which specifically called for legislation on 
the liability of robots and AI (Resolution on Civil Law Rules on Robotics, 
2017). Furthermore, the Commission adopted “Communication on Artificial 
Intelligence for Europe” on April 25, 2018 (Communication on Artificial 
Intelligence for Europe, 2018,). With the help of an expert panel, the 
Commission stated in this communication that it will examine if the national 
and EU liability frameworks are appropriate in the context of problems posed 
by AI. Two years later, the Commission published a package consisting of four 
documents,including the White Paper “On Artificial Intelligence -—A European 
approach to excellence and trust” (Koch, 2020). In April 2021 European 
Commission moved ahead with the Proposal for the Artificial Intelligence Act 
(hereinafter: AIA proposal), which will present the main subject of the research 
in the paper (Proposal for the Artificial Intelligence Act, 2021). 

The AIA proposal is the first initiative to horizontally regulate AI on 
a global level (Bogucki, Engler, Perarnaud & Renda, 2022). It establishes 
fundamental, cross-industry norms for the creation, exchange, and application 
of Al-driven systems, products, and services within EU territory. This act aims 
to formalize the high requirements of the “Ethics guidelines for trustworthy 
Al(a)”, which calls for AI to be technically proficient, ethical, and lawful 
while safeguarding democratic principles, human rights, and the rule of law 
(Hickman & Petrin, 2021). In order to meet this aim, the AIA proposal follows 
a risk-based approach to differentiate between AI systems uses that create 
the following categories of risks: “an unacceptable risk, a high risk,and a 
low or minimal risk” (Explanatory Memorandum of the AIA proposal, 2021, 
p.12).This implies, among other things, that applications using AI that pose 
an unacceptable risk are prohibited, while AI systems with low risks, can be 
created and used in compliance withcurrent regulations. 
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Considering the abovementioned, the goal of the article is to move 
toward a possible resolution to the dilemma of whether the AIA proposal is 
appropriate for the AI era by addressing the scope of this act, the prohibited 
AI practices, rules on high-risk AI systems, specific transparency obligations 
as well as certain regulatory gaps. 


2. The scope of the AIA proposal 


The scope of the AIA proposal is defined by the subject matter of the 
regulation as well as the scope of its application. Concerning the subject 
matter, Article 1 states thatthe AIA proposal establishes: 

(a) “harmonised rules for the placing on the market, the putting into 
service and the use of artificialintelligence systems (‘AI systems’) in 
the Union; 

(b) prohibitions of certain artificial intelligence practices, 

(c) specific requirements for high-risk AI systems and obligations for 
operators of such systems, 

(d) harmonised transparency rules for AI systems intended to interact 
with natural persons, emotionrecognition systems and biometric 
categorisation systems, and AI systems used to generate ormanipulate 
image, audio or video content; 

(e) rules on market monitoring and surveillance” (Proposal for Artificial 
Intelligence Act, 2021). 


According to Article 1, the AIA proposal regulates “AI systems”. 
Along with the issue of how to distinguish between “AI” and “AI systems”, 
theextremely broad conceptual scope of the AlAproposal also looks 
unclear. The definition of “AI systems” is provided by Article 3(1) of the 
AIA proposal, which together with Annex I mainly includes any computer 
program. As a result of such a wide approach, the designers, operators, and 
users of AI systems may experience different legal uncertainty(Helberger & 
Diakopoulos, 2022).Undoubtedly, a broad definition of “AI systems” may be 
reasonable in the context of the AI practices expressly forbidden by Article 5 
of the AIA proposal in order to balance the risks that various types of software 
pose tofundamental human rights. Contrary, when it concerns high-risk AI 
systems, such a broad definition is too general. The required conditions 
proposed within Title HI of the AIA proposal for these systems are based on 
the understanding that many fundamental rights are negatively affected by 
the unique features of machine learning, including transparency, complexity, 
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reliance on data, and autonomous behaviour (Smuha et al., 2021, p. 11). The 
wide definition of AI may result in overregulation because these features are 
either not present or only partially present in simple algorithms (Ebers, Hoch, 
Rosenkranz, Ruschemeier & Steinrétter, 2021, p. 591). 

In regards to the territorial scope, the AIA would apply to public and 
commercial actors both inside and outside the EU, so long as their AI system 
is sold on the EU market or has an impact on EU citizens. The AIA wouldapply 
to three types of companies (or other parties, including public bodies), that use 
AI systems in different ways: providers, users, and producers of products used 
in the EU. The first and third categories, give the AIA proposal extraterritorial 
impact outside of the EU (Greenleaf, 2021, p. 3).By restricting the geographic 
application of the AIA proposal to the “use” of AI systems within the EU, it 
is possible that some high-risk AI systems or even forbidden AI systems are 
developed, sold, or exported from the EU but used outside the EU. Therefore, 
it seems that this provision has the potential to create various legal and ethical 
problems for users of AI systems outside the EU (Ebers, Hoch, Rosenkranz, 
Ruschemeier, & Steinrétter, 2021, p. 591). 


3. Prohibited uses of AI 


Article 5 of the AIA proposal establishes a list of prohibited AI practices. 
The list of prohibited practices includes all AI systems whose use is not in 
accordance with fundamental European values, suchas respect for fundamental 
human rights and freedoms.Four different types of AI are generally included 
under the list ofAI practices that are prohibitedunder the standards outlined in 
Article 5 of the AIA proposal. 

The first one, “subliminal or manipulative AI practices”, is defined 
as one that has “a significant potential to manipulate persons through 
subliminal techniquesbeyond their consciousness” to materially modify 
someone’sbehaviour in a way that harms or is likely to negatively affect their 
physical or psychological well-being or the well-being of another person 
(Explanatory Memorandum of the AJA proposal, 2021, p.12). Even though 
the AIA proposal does not define the term “subliminal”, this phrase typically 
describes a perception that is below the level of awareness (Klein, 1966, p. 
726). The activity’s potential to harm someonephysically or psychologically 
should be considered a final trigger. The scope of the provision is significantly 
limited by this requirement(Veale & Borgesius, 2021 p. 99). 

The second type of prohibited AI is referring to the AI practices 
exploiting vulnerabilities of particularly vulnerable groups including children 
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or persons with disabilitiesto materially influence a person’s behaviour in a 
way that harms or is likely to harm that person or another person’s physical or 
psychological well-being.The main aspect of this provision is vulnerability, 
which isnot extensively defined but only demonstrated by the examples of 
particularly vulnerable groups, such as children or individuals with disabilities 
(Neuwirth, 2022, p. 7). 

The thirdcategory of prohibited AI practices, “social scoring systems”, 
includes systems used by “public authorities or ontheir behalf for the 
evaluation or classification of the trustworthiness of natural personsover 
a certain period of time-based on their social behaviour or known or 
predictedpersonal or personality characteristics” (Article 5 of the Proposal 
for the Artificial Intelligence Act, 2021). It seems that by restricting the use 
of social scoring to public authorities, the AIA proposal ignores the use of 
such systems by private businesses, especially in high-risk sectors where 
they may have the potential to indirectly impactfundamental rights. Various 
infrastructures including delivery, telecommunications, and transportation are 
under the authority of so-called AI companies (Rahman, 2017). Therefore, the 
above exclusion can have serious socioeconomic implications for individuals, 
which imposes the needto make this provision universally applicable. 

The use of “real-time remote biometric identification systems in publicly 
accessible locations” falls under the fourth category of prohibited AI practices 
with exception of certain law enforcement reasons (Article 5 of the Proposal 
for the Artificial Intelligence Act, 2021).The Law Enforcement Directive 
(Directive (EU) 2016/680),regulates the use of biometric identification for 
law enforcement purposes.The widely accepted critics of the doctrine are 
referring to the narrow scope and limitation of lawenforcement thatallows 
the use of such AI systems for different purposes (Gill-Pedro, 2021).The 
use of remote biometric identification for non-law enforcement objectives 
like crowd control or public health is not prohibited by the restriction. 
The GDPR normally applies to these uses (Regulation (EU) 2016/679). In 
general, the GDPR imposes a criterion of high-quality, individual permission 
for each person scanned, which is practically hard to provide in the absence 
of a corresponding Member State law authorizing such biometrics (Veale & 
Borgesius, 2021, p. 101). 

In addition, the fact that Article 5 could not be amended by the European 
Commission could be quite challenging in the context of the implementation 
of the AIA due to the fact that some problematic aspects of AI practices can 
only be recognized ex-post. 


36 


HUMAN VS. ARTIFICIAL INTELLIGENCE — EU’S LEGAL RESPONSE 


4. Rules on high-risk AI systems 


For AI systems that create a high risk to human health and safety 
or fundamental rights, or “high-risk AI systems,” Title UI of the AIA 
proposalestablishes a new regulatory regime with precise standards. The AIA 
Proposal adopts a prescriptive “list-based approach,” which outlines which 
systems are considered a high risk rather than defining the term itself.Based 
on the AI system’s intended use and current product safety regulations, a 
system is categorized as high-risk. As a result, the categorization of a high- 
risk depends not only on the task performed by the AI systembut also on the 
precise objectives and operating procedures of that system. 

Two main groups of high-risk AI systems are identified in Title II, along 
with the classification criteria. Systems intended for use as safety components 
of products that are covered by “third-party ex-ante conformity assessment” 
under EU law are included in Annex II of the proposal as high-risk systems, 
as are other standalone AI systems used in high-risk domains (Explanatory 
Memorandum of the AIA proposal, 2021, p. 14).The European Commission 
has identified eight use categories for high-risk standaloneAI systems listed 
in Annex III. By using a set of criteria and a risk assessment methodology, 
the European Commission may expand the list of high-risk AI systems used 
within specified pre-defined sectors in order to ensure that the legislation may 
be modified to develop uses and applications of Al.However, it is important 
to note that the Commission can only do this if the high-risk AI systems are 
intended to be used in any of the activities stated in Annex III points 1 through 
8. This provision could be quite challenging due to the fact, that we cannot be 
aware of all categories of high-risk systems sinceAI is a rapidly evolving field 
that is progressively influencing other industries (Smuha et al., 2021, p. 11). 

In addition, Chapter 2 outlines the legal requirements for high-risk AI 
systems related to “data and data governance, documentation and recording 
keeping, transparency and provision of information to users, human 
oversight, robustness, accuracy and security” which links to obligations 
of regulated actors stated within Chapter 3 (Explanatory Memorandum of 
the AIA proposal, 2021, p. 13). The great majority of all obligations are 
theresponsibility of providers.With respect to data and data governance, 
Article 10 of the AIA proposal mostly refers to training, validation, and 
testing data sets. Data quality criteria for sets of data on individuals, or groups 
of people (not necessarily involving personal data in GDPR terms), including 
“special categories of personal data” (as defined in Article 9 of GDPR) are 
highly detailed in the subject requirements (Regulation (EU) 2016/679). 
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The following requirement is referring totechnical documentation. Providers 
must submit technical documentationthatincludes all information in line with 
Annex IV. Moreover,according to Article 12 of the AIA proposal and record- 
keeping requirements, providers need to facilitate logging in order to enable 
traceability that is acceptable for a system’s risks.Providers are only required 
to keep logs for the relevant period while such logs are still under their control; 
otherwise, users are required to do so.The standards for high-risk AI systems 
transparency are defined in Article 13. A high-risk AI system must be created 
in accordance with Article 13 in order to be “sufficiently transparent to enable 
users to interpret the system’s output and use it appropriately” and it must 
also come with instructions and information that are “relevant, accessible, 
and comprehensible to users” (Article 12 of the Proposal for the Artificial 
Intelligence Act, 2021). In addition to the standards above, Article 14 
stipulates that providers must create systems that can be properly supervised 
by natural persons, using “human-machine interface tools” (Article 14 of the 
Proposal for the Artificial Intelligence Act, 2021). To ensure the protection of 
fundamental rights, oversight is necessary for all actions linked to the creation, 
implementation, and use of AI systems.Moreover, Article 15 states that high- 
risk AI systems must be created and constructed in such a way that, in the 
context of their intended use, they achieve the required level of accuracy, 
robustness, and cybersecurity and operate consistently over the period of their 
lifecycle (Article 15 of the Proposal for the Artificial Intelligence Act, 2021). 
The framework for notified bodies’ participation in conformity assessment 
processes as independent third parties is provided in Chapter 4, while the specific 
conformity assessment processes that must be implemented for every type of 
high-risk AI system are included in Chapter 5. The approach to conformity 
assessment aims to reduce pressure on both notified entities and economic 
operators, whose capability must be gradually ramped up through time. 


5. Specific transparency obligations 


Title IV of the AIA proposal outlinesspecific transparency obligations. 
The AIA proposal introduces transparency requirements for systems 
thatinteract with humansdue to the fact thatpeople have a right to know when 
they are engaging with a machine’s algorithm rather than a human being. 
Similar requirements for transparency apply to the disclosure of deep fake/ 
synthetics, biometric categorization, and automated emotion detection 
systems.Except for biometric categorization systems that are legally allowed 
to be used for crime prevention, users of emotion recognition or biometric 
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categorization systems are required to notify exposed persons of the system’s 
operation. In comparison with data protection law, it is quite challenging to 
understand the contribution of this provision. Data protection law indicates 
that users of emotional recognition or biometric categorization systems that 
process personal datanotify individuals of, among other things, the existence 
and purposes of such processing.Therefore, it is difficult to determine what is 
the real scope of this provision. 

In addition, specific transparency obligations are also introduced for 
limited-risk AI systems like chatbots. The Low-Risk AI Systems category is 
the only one that is excluded from transparency obligations (Kop, 2021). 


6. Identifying additional regulatory gaps of the AIA proposal 


Even though the above analysis of the AIA proposal has already 
identified certain aspects of the Act that need further clarification, the doctrine 
concluded that this act has some additional gaps. The most significant one 
is referring to the fact that the AIA proposal does not include any individual 
right of enforcement. Although the Act is designed to protect fundamental 
rights, it has no remedies through which individuals can seek redress if the 
regulation is violated. The AIA proposal does not include any mechanism to 
allow individuals to challenge AI-driven decision-making (Ebers, 2021 p. 19). 

Moreover, a European approach to AI, on the other hand, should 
consider not only human rights but also other priorities such as climate 
change and sustainability. In this respect, the AIA proposal makes no direct 
mention of “Green AI” or “Sustainable AI” as a clear objective of a European 
understanding of AI development according to the standards of the European 
Green Deal (Gailhofer et al. 2021). The Act only recognizes the necessity for 
relevant action in the high-impact field of climate change and the potential of 
AI to help socially and environmentally positive outcomes. 


7. Conclusion 


The AIA proposal intends to establish a uniform legal system for AI in the 
EU. Through a comprehensive framework, the AIA proposal addressesboth the 
potential benefits of AI and the moral questions raised by the different threats 
associated with it.Nevertheless,some aspectsrequirefurther clarification. The 
main aspect that needs to be improved is the definition of the term “AI”. The 
AIA proposalincludes a quite broad definition, which increases the risk of 
overregulation of systems. Furthermore, the lack of individual enforcement 
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rights in the AIA proposal underminesthe protection of fundamental rights as 
the most important goal of this regulation. The AIA must guarantee the right 
to remedy that addresses potential Regulation violations or infringements of 
fundamental rights. 

This article cannot and has not discussed all aspects of the AIA proposal. 
The author has demonstrated some of the complexities of this particularly 
significant instrument. After all, creating a safe and adequate regulatory 
framework for AI in Europe is not only the way we design technology but 
also the way we shape our society’s future. 


Mladenov Marijana 
Pravni fakultet za privredu i pravosude u Novom Sadu, Univerzitet Privredna akademija 
u Novom Sadu, Srbija 


LJUDSKA PROTIV VESTACKE 
INTELIGENCIJE — PRAVNI ODGOVOR EU 


REZIME: VeStacka inteligencija ima kapacitet da poboljSa ne samo kvalitet 
zivota pojedinca, veé i ekonomsko i socijalno blagostanje. Iako sistemi 
veStatke inteligencije imaju mnoge prednosti, oni takode predstavljaju 
znaéajne rizike, stvarajuci Sirok spektar moralnih i pravnih dilema. Evropska 
unija kreira pravni okvir za razvoj, trgovinu i upotrebu proizvoda, usluga 
isistema vodenih veStackom inteligencijom kako bi smanjila rizike povezane 
sa sistemima veStactke inteligencije i sprecila svaku mogucu Stetu koju oni 
mogu da izazovu. Glavni fokus ovog rada odnosi se na analizu Predloga 
Uredbe 0 veStackoj inteligenciji koji je Evropska komisija podnela u aprilu 
2021. Cilj élanka je da pruzi doprinos u kontekstu razreSenja dileme da li 
je predlog navedene uredbe adekvatan zahtevima ere veStatke inteligencije, 
adresirajuci obim primene ovog akta, zabranjene prakse veStatke 
inteligencije, pravila o visokoriziénim sistemima veStacke inteligencije, 
specifiéne obaveze transparentnosti kao i odredene pravne praznine. 
Clanak treba posmatrati kao poéetnu analizu predloga Uredbe 0 veStaékoj 
inteligenciji kako bi se obezbedio koristan okvir za buducu diskusiju. 


Kljuéne reci: vestacka inteligencija, Evropska unija, regulatorni okvir, 
Predlog Uredbe o vestackoj inteligenciji. 
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